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EVALUATION  OF  THE  VIRTUAL  SQUAD  TRAINING  SYSTEM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

At  the  request  of  TRADOC’s  Program  Integration  Office  (TPIO)  Virtual,  National  Simulation 
Center  at  Ft.  Leavenworth,  personnel  from  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  Orlando  Research  Unit  participated  in  an  evaluation  of  the  Virtual 
Squad  Training  System  (VSTS)  from  May  7  to  1 1  2007.  The  VSTS  is  located  in  the  Battle 
Command  Training  Center  (BCTC),  at  Schofield  Barracks,  Hawaii. 

The  VSTS  incorporates  nine  individual  simulators  so  that  each  member  of  a  squad  can  be 
immersed  in  a  common  Virtual  Environment  (VE).  Five  of  the  simulators  are  freestanding 
“wearable”  simulators  using  wireless  networking.  Four  are  “tethered”  with  an  overhead  cable 
connecting  to  a  computer  system.  Both  types  of  simulators  include  Helmet  Mounted  Displays 
(HMDs),  weapon  mockups,  and  motion  control  interfaces  so  that  the  users  (trainees)  can  move 
and  shoot  within  the  shared  VE.  The  VSTS  also  includes  a  BattleMaster  station,  a  Semi- 
Automated  Force  (SAF)  workstation,  a  Scenario  Authoring  station,  and  an  After  Action  Review 
station.  SAF  was  used  to  populate  the  VE  with  computer  generated  opposing  forces,  friendly  and 
neutral  forces,  and  civilian  pedestrians  and  vehicles.  The  VE  used  for  this  evaluation  was  a  geo¬ 
typical  database  of  Northern  Baghdad,  Iraq. 

Procedure: 

Two  squads  of  Soldiers  performed  selected  individual/fire  team  tasks  and  squad  tactical 
exercises  over  a  four  day  period  (two  days  per  squad).  The  evaluation  plan  called  for  the  Soldiers 
to  be  assigned  to  either  a  tethered  or  wearable  system  for  the  first  day,  and  then  switched  to  the 
other  type  of  system  on  the  second  day. 

The  morning  of  their  first  day  of  participation,  the  Soldiers  received  an  orientation  briefing,  a 
familiarization  session  with  the  system,  and  structured  train-up  and  practice  with  the  simulators. 
For  the  afternoon  of  the  first  day  and  most  of  the  second  day  the  Soldiers  conducted  squad  level 
exercises  in  the  VE  simulators. 

The  evaluation  included  the  following: 

•  Background  information  survey  regarding  previous  relevant  training  and  experience. 

•  Soldier  self-rating  of  ability  to  perform  62  basic  functions  in  the  simulator. 

•  Soldier  self-rating  of  the  training  effectiveness  of  the  VE  exercises  conducted  with  the 
simulator  that  day. 

•  Soldier  self-rating  of  symptoms  of  simulator  sickness. 
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•  A  structured  exit  interview  that  addressed  more  than  a  dozen  issues  ranging  from  the 
usability  of  the  individual  simulators  to  the  potential  training  applieations  of  the  overall 
system.  Separate  interviews  were  condueted  for  each  squad,  and  within  each  squad 
separate  interviews  were  conducted  with  the  squad  leader  and  two  team  leaders  together 
and  the  other  squad  members  together. 

Frequent  teehnical  problems  with  the  system  and  some  of  the  individual  simulators  interfered 
with  conducting  the  evaluation  as  planned.  More  importantly,  teehnieal  problems  undoubtedly 
influenced  almost  every  aspect  of  the  Soldiers’  ratings  of  the  system  components,  and  the  system 
as  a  whole. 

Findings: 

Soldier  Characteristies 

Each  squad  had  nine  Soldiers  with  an  appropriate  MOS  (11  Bravo).  Both  squad  leaders  and  all 
the  fire  team  leaders  had  previous  experienee  at  their  respective  positions  in  a  dismounted 
infantry  squad. 

Simulator  Sickness 

Soldier  self-rating  of  symptoms  of  simulator  sickness  included  16  items  associated  with 
simulator  sickness.  The  post  exposure  scores  were  significantly  (in  the  statistical  sense)  higher 
than  the  pre-exposure  baseline  scores,  but  eompared  to  previous  research  with  HMDs  the 
frequency  and  severity  of  simulator  sickness  symptoms  were  very  low.  No  one  withdrew  from 
the  evaluation  beeause  of  simulator  sickness.  The  predominant  symptoms  were  related  to  eye 
strain.  There  were  no  major  problems  with  nausea  or  dizziness,  whieh  are  the  symptom 
categories  that  are  more  problematic  from  a  safety  perspective. 

Simulator  Performance  Questionnaire 

Soldiers  completed  a  questionnaire  rating  their  ability  to  perform  62  basic  functions  in  the 
simulator.  Examples  are:  aim  weapon,  move  as  an  individual,  visually  locate  the  source  of 
enemy  fire,  determine  the  source  of  enemy  fire  by  sound,  and  communieate  enemy  loeation  to 
team  member.  Six  of  the  items  had  an  average  rating  above  “good”.  Two  of  these  involve 
movement  and  four  involve  use  of  the  weapon.  Eifty-two  items  were  rated  between  “average” 
and  “good”.  Three  items  had  mean  ratings  between  “average”  and  “poor.”  The  three  lowest 
rated  items  were  “Determine  the  souree  of  enemy  fire  by  sound.”,  “Open  gates”,  and  “Open 
doors”.  The  lack  of  realistic  three  dimensional  sound  was  mentioned  during  the  interviews  as  a 
system  shortfall.  However,  in  the  real  world  location  of  enemy  fire  by  sound  in  urban  areas  is  a 
well  known  ehallenge.  So  in  this  case  the  low  rating  may  not  represent  that  big  of  a  problem.  In 
eontrast,  the  low  ratings  for  opening  doors  and  gates  do  indieate  problems  that  need  to  be 
resolved. 
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Exercise  Training  Effectiveness 


A  questionnaire  was  used  to  address  Soldier  self-ratings  of  the  training  effectiveness  of  the 
exercises  conducted  with  the  simulator  that  day.  The  Soldiers  rated  the  change,  if  any,  in  their 
ability  to  perform  24  tasks.  Examples  are;  employ  hand  grenades,  react  to  direct  and  indirect  fire, 
and  react  to  a  civil  disturbance. 

The  highest  rated  task  was  “perform  voice  communications”.  The  average  rating  of  .97  falls 
slightly  below  “1”  the  value  that  corresponds  to  “slight  improvement”.  In  general  these  ratings  of 
training  effectiveness  are  low.  We  believe  the  ratings  are  a  function  of  several  factors.  The 
scenarios  were  good;  however  the  exercises  and  After  Action  Reviews  were  greatly  hindered  by 
technical  difficulties.  There  is  also  another  critical  factor  to  the  ratings.  Most  of  the  Soldiers  who 
participated  believed  that  they  already  knew  how  to  perform  these  tasks  before  they  took  part  in 
this  evaluation.  Therefore,  from  their  perspective  there  was  little  room  for  improvement. 

Interview  Comments 

A  structured  interview  was  used  to  address  more  than  a  dozen  different  issues.  Throughout  the 
interviews,  regardless  of  the  specific  questions,  there  were  frequent  comments  that  the  system 
was  frustrating  to  use,  that  the  “bugs”  should  be  fixed,  and  more  practice  was  needed  on  how  to 
use  the  system.  They  made  it  clear  that  by  more  practice  they  meant  additional  days,  not  just  a 
few  more  hours. 

Eor  performance  and  ease  of  use  the  tethered  simulator  was  clearly  preferred  to  the  wearable 
simulator.  Better  HMD  resolution  and  less  lag  were  cited.  The  speed  and  ease  of  getting  in  and 
out  of  the  system  was  also  praised.  However,  there  were  two  common  complaints  about  the 
tethered  design.  The  HMD  fit  on  the  nose  was  not  good,  padding  and  perhaps  an  eye  cup  are 
needed.  Also,  the  cable  seemed  “stiff’  during  turns  and  was  awkward  for  the  tallest  users. 

The  Soldiers  liked  two  aspects  of  the  wearable  system.  The  design  of  the  HMD  connection  to  the 
helmet  was  praised  as  very  comfortable.  Also,  it  was  easy  to  turn  360  degrees.  There  were 
several  complaints  about  the  “leg  sensor”  (the  leg  worn  body  posture  tracker).  It  was  difficult  to 
keep  adjusted  properly. 

Several  aspects  of  the  VE  database  were  cited  as  good.  The  expanse  of  the  database  provided 
room  to  maneuver.  The  look  of  the  buildings,  and  the  pervasive  people  and  trash  were  realistic. 

There  were  several  problems  cited  in  controlling  movement.  It  was  difficult  to  move  close  to  or 
around  obstacles  inside  a  building.  Their  avatar  was  frequently  running  when  they  were  trying  to 
walk. 

Other  problems  cited  included  the  static  OPEOR  and  something  very  wrong  in  the  representation 
of  moving  vehicles  (approaching  vehicles  did  not  make  appropriate  sounds  and  occasionally 
collided  with  pedestrians).  Difficulty  seeing  distant  objects  and  the  lack  of  peripheral  vision 
were  cited.  Keeping  the  visual  representation  of  the  weapon  “up”  was  a  big  problem.  Many  of 
the  menu  functions  took  too  long  to  perform. 
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The  interviewees  felt  that  if  the  bugs  were  fixed  the  system  might  be  useful  for  an  introduction 
for  new  Soldiers  and  for  providing  training  for  small  team  leaders  lacking  experience.  The 
system  would  not  be  useful  for  sustainment.  The  trainees  did  not  believe  that  the  VSTS  was 
suitable  for  use  in  the  field.  If  a  VE  of  an  operational  area  were  available,  it  would  be  very 
useful  for  mission  rehearsal.  It  was  not  clear  that  they  would  want  to  view  the  terrain  model 
using  the  HMDs,  but  a  detailed  VE  would  of  course  be  of  value. 

Einally,  the  Soldiers’  responses  when  asked  for  their  overall  reaction  to  the  VSTS  were:  “Has 
potential,  but  needs  a  lot  of  work”,  “Could  be  useful  if  fixed”. 

Utilization  and  Dissemination  of  bindings: 

The  results  were  briefed  to  TPIO  in  Jun  07.  The  results  can  be  used  in  decisions  on  the 
development  and  use  of  virtual  training  systems. 

Erequent  technical  problems  with  the  system  and  some  of  the  individual  simulators  interfered 
with  conducting  the  evaluation  as  planned.  More  importantly,  technical  problems  undoubtedly 
influenced  almost  every  aspect  of  the  Soldiers’  ratings  of  the  system  components,  and  the  system 
as  a  whole. 

Some  of  the  problems  are  with  the  software,  some  are  hardware  related,  and  some  are  the  results 
of  inadequate  train-up  for  the  user.  Some  problems  may  result  from  interactions  of  all  three.  To 
the  Soldiers  trying  to  conduct  exercises  with  the  system  the  sources  of  the  problems  are  not  clear; 
however,  it  is  clear  that  they  are  frequently  frustrated  in  trying  to  use  the  system. 

If  the  system  performance  is  not  going  to  be  improved  over  the  level  we  observed,  then  the 
usefulness  of  the  system  is  in  question  because  training  effectiveness  would  be  compromised  by 
the  various  system  problems. 
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EVALUATION  OF  THE  VIRTUAL  SQUAD  TRAINING  SYSTEM 


INTRODUCTION 

This  report  describes  an  evaluation  of  the  Virtual  Squad  Training  System  (VSTS)  conducted 
from  7  to  1 1  May  2007  at  Schofield  Barracks,  HI.  The  evaluation  was  lead  by  TRADOC’s 
Program  Integration  Office  (TPIO)  Virtual,  National  Simulation  Center  at  Ft.  Leavenworth,  KS. 
The  VSTS  is  located  in  the  Battle  Command  Training  Center  (BCTC),  at  Schofield  Barracks,  HI. 
The  evaluation  was  specifically  scheduled  to  take  advantage  of  the  availability  of  two  infantry 
squads  from  2d  BCT,  25*  ID. 

The  rest  of  the  report  is  organized  as  follows.  A  brief  background  section  is  presented  with 
references  to  much  more  highly  detailed  history  of  development  and  use  of  virtual  simulators  for 
dismounted  infantry  training.  The  evaluation  procedure  is  described.  Results  are  listed  and 
discussed.  A  summary  and  conclusion  are  presented. 


BACKGROUND 

Largely  as  a  result  of  the  influence  of  Gorman  (1990),  an  early  proponent  of  the  use  of  virtual 
environments  (VE)  for  dismounted  infantry  (DI)  training,  the  U.S.  Army  began  to  be  interested 
in  the  use  of  immersive  simulation  for  dismounted  Soldier  training  in  the  early  1990’s.  As  for 
other  training  domains,  goals  for  using  simulation  for  DI  training  include  providing  a  cost 
effective,  safe,  and  flexible  complement  to  live  training. 

The  Dismounted  Warrior  Network  (DWN)  effort  was  a  U.S.  Army  Simulation,  Training  and 
Instrumentation  Command  (STRICOM)  program  to  develop  a  reliable,  low-cost,  easy-to-use 
capability  to  insert  dismounted  Soldiers  into  VE.  The  DWN  program  (and  its  successor,  DWN 
Enhancements  for  Restricted  Terrain  (ERT))  did  not  evaluate  training  effectiveness,  but  did 
obtain  data,  through  automated  data  capture.  Soldier  questionnaires,  and  observation,  about  task 
performance  in  the  virtual  simulators  used  in  the  program.  Tasks  covered  movement,  orientation, 
visual  recognition,  and  weapon  engagement.  Detailed  descriptions  of  these  efforts  are  presented 
in  Lockheed  Martin  (1997)  and  Pleban,  Dyer,  Salter,  and  Brown  (1998). 

Following  the  completion  of  the  DWN  experiments,  a  four-year  (FY  1999-2002)  Science  and 
Technology  Objective  (STO)  effort  was  initiated  to  develop  a  demonstration  virtual  dismounted 
leader  trainer  at  the  fire  team,  squad,  and  platoon  level.  Each  year  of  the  STO  a  culminating 
event  (CE)  was  conducted.  The  CEs  were  comprehensive  demonstrations  and  assessments 
conducted  with  Soldiers,  using  as  much  of  the  developed  technology  as  was  feasible  in  a  realistic 
training  exercise.  This  body  of  research  is  described  by  Knerr,  et  al.(2003). 

The  Virtual  -Integrated  Military  Operations  in  Urban  Terrain  (MOUT)  Training  System  (V- 
IMTS)  was  a  short-term  project  to  speed  the  transition  of  the  virtual  simulation  technology 
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developed  under  the  STO  that  speeifically  eonsidered  the  integration  of  live  and  virtual  training. 
The  objective  of  the  assessment  was  to  obtain  information  about  the  performance  of  Soldier  tasks 
and  training  effectiveness  in  the  V-IMTS  configuration.  The  Cassidy  Combined  Arms  Collective 
Training  Center  at  Ft.  Campbell,  KY,  was  the  site  for  the  assessment  (Knerr  &  Lampton,  2005). 

Throughout  this  decade  of  research,  the  Soldier  participants  have  identified  functions  they  felt 
were  needed  to  improve  the  potential  training  effectiveness  of  immersive  DI  training  systems. 
Successive  VE  systems  have  incorporated  many  of  these  functions,  and  our  data  collection 
instruments  have  been  modified  to  address  these  added  functions. 


METHOD 

The  VSTS  System. 

The  VSTS  is  located  in  the  BCTC,  at  Schofield  Barracks,  HI.  The  BCTC  is  a  modem  90,000- 
square-foot,  two-story  training  center  with  excellent  spatial  layout,  lighting,  acoustics,  and 
temperature  control. 


The  VSTS  (described  in  detail  in  Appendix  A)  incorporates  nine  individual  simulators  so  that 
each  member  of  a  squad  can  be  immersed  in  a  common  VE.  Eive  of  the  simulators  are 
freestanding  “wearable”  simulators  using  wireless  networking  (see  Eigure  1).  Eour  are 
“tethered”  with  an  overhead  cable  connecting  to  a  computer  system  (see  Eigure  2).  Eigure  1 
conveys  how  the  nine  squad  members  were  co-located  during  the  exercises.  Also  shown  are  the 
weapon  mockups  for  the  M4,  the  M203,  and  the  M249. 


Eigure  1.  Eour  Soldiers  using  the  wearable  simulator. 
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Figure  2.  A  Soldier  using  the  tethered  simulator. 


Both  types  of  simulators  include  Helmet-Mounted  Displays  (HMDs),  weapon  mockups,  and 
motion  control  interfaces  so  that  the  users  (trainees)  can  move  and  shoot  within  the  shared  VE. 
(During  the  evaluation  the  VSTS  intra-squad  communication  was  inoperable.  However,  because 
all  nine  simulators  were  co-located  the  squad  members  could  communicate  by  simply  yelling  at 
each  other.) 

The  VSTS  also  includes  a  BattleMaster  station,  a  Semi-Automated  Force  (SAP)  workstation,  a 
Scenario  Authoring  station,  and  an  After  Action  Review  (AAR)  station.  SAF  was  used  to 
populate  the  VE  with  computer  generated  opposing  forces,  friendly  and  neutral  forces,  and 
civilian  pedestrians  and  vehicles  (see  Figure  3).  The  VE  used  for  this  evaluation  was  a  geo¬ 
typical  database  of  Northern  Baghdad,  Iraq. 
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Figure  3.  Screen  shot  from  an  exercise.  (The  avatar  “BEC”  is  controlled  by  a 
Soldier  in  a  simulator.  The  other  avatars,  including  vehicles,  are  computer 
controlled  entities  generated  and  controlled  from  SAP  station.). 

Procedure 

Two  squads  of  Soldiers  performed  selected  individual/fire  team  tasks  and  squad  tactical 
exercises  over  a  four  day  period  (two  days  per  squad).  The  evaluation  plan  called  for  the  Soldiers 
to  be  assigned  to  either  a  tethered  or  wearable  system  for  the  first  day  and  then  switched  to  the 
other  type  of  system  on  the  second  day.  (A  detailed  description  of  the  planned  procedure  is  in 
Appendix  B.  The  actual  procedure  followed  is  reported  below.) 

The  morning  of  their  first  day  of  participation,  the  Soldiers  received  an  orientation  briefing,  a 
familiarization  session  with  the  system,  and  a  structured  train-up  and  practice  with  the 
simulators.  For  the  afternoon  of  the  first  day  and  most  of  the  second  day  the  Soldiers  conducted 
squad  level  exercises  in  the  VE  simulators. 

Available  exercises: 

•  Break  contact 

•  React  to  ambush 

•  Combat  Patrol 

•  Clear  a  building/room 

•  Call  for  and  receive  QRF  support 

•  Call  for  MEDIVAC 

•  Establish  a  hasty  observation  point 

Numerous  technical  problems  made  it  impossible  to  keep  track  of  the  actual  scenarios  that  were 
successfully  completed  or  the  amount  of  time  that  the  Soldiers  spent  training  with  those 
scenarios.  As  a  very  rough  estimate,  we  believe  that  the  first  squad  spent  a  maximum  of  2  hours, 
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53  minutes  in  nine  scenarios,  while  squad  two  spent  a  maximum  of  1  hour,  24  minutes  in  seven 
scenarios. 

The  evaluation  included  the  following; 

•  Background  information  survey  regarding  previous  relevant  training  and  experience. 

•  Soldier  self-rating  of  ability  to  perform  62  basic  functions  in  the  simulator. 

•  Soldier  self-rating  of  the  training  effectiveness  of  the  virtual  environment  exercises 
conducted  with  the  simulator  that  day. 

•  Soldier  self-rating  of  symptoms  of  simulator  sickness. 

•  A  structured  exit  interview  that  addressed  more  than  a  dozen  issues  ranging  from  the 
usability  of  the  individual  simulators  to  the  potential  training  applications  of  the  overall 
system.  Separate  interviews  were  conducted  for  each  squad,  and  within  each  squad 
separate  interviews  were  conducted  with  the  squad  leader  and  two  team  leaders  together 
and  the  other  squad  members  together. 

As  mentioned  above,  frequent  technical  problems  with  the  system  and  some  of  the  individual 
simulators  interfered  with  conducting  the  evaluation  as  planned.  More  importantly,  technical 
problems  influenced  almost  every  aspect  of  the  Soldiers’  ratings  of  the  system  components  and 
the  system  as  a  whole. 

Before  the  evaluation,  an  entire  day  was  allocated  to  finalizing  the  scenarios;  however  frequent 
system  crashes  interfered  with  setting  up  the  scenarios  and  actually  testing  the  scenarios  with 
participants  immersed  in  the  individual  simulators.  Although  the  scenarios  were  in  general  well 
conceived,  the  effectiveness  of  many  of  the  scenarios  was  compromised  because  of  the  inability 
to  make  final  adjustments  such  as  to  the  placement  of  Opposing  Forces  (OPFOR)  and 
Improvised  Explosive  Devices  (lEDs). 

The  system  experienced  several  different  types  of  technical  problems.  The  network  had  frequent 
crashes.  The  individual  simulators  frequently  required  some  adjustment  by  system  support 
personnel.  The  individual  simulators  frequently  required  a  relatively  brief  and  simple 
recalibration  procedure  that  could  be  performed  by  the  user.  The  amount  of  disruption  caused  by 
the  self  calibration  was  a  function  of  when  it  occurred  during  an  exercise.  If  it  was  required 
during  the  middle  of  a  fire  fight  then  it  was  highly  disruptive,  if  it  was  required  during  a  halt  it 
was  a  negligible  inconvenience.  Delays  in  starting  the  exercises  and  problems  during  the 
exercises  were  frustrating  for  the  Soldiers  participating  in  the  evaluation. 

The  AAR  system  was  occasionally  not  available  after  an  exercise,  it  sometimes  crashed  during 
an  AAR,  and  at  least  once  it  could  not  show  a  critical  incident  to  determine  what  had  happened 
(a  wall  occluded  identification  of  which  squad  member  had  fired  a  shot).  The  research  planned 
called  for  the  VSTS  AAR  application  to  be  used  to  capture  statistical  data  such  as  number  of 
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rounds  fired,  number  of  hits,  movement  routes,  movement  teehniques,  communications 
transmissions,  and  target  identification.  However,  none  of  these  functions  were  in  evidence 
during  the  evaluation. 

Additional  problems  resulted  from  the  illness  of  one  of  the  two  support  personnel.  For  an  entire 
day  and  part  of  another  the  system  operator  was  absent  because  of  illness.  Although  the  other 
support  person  did  an  outstanding  job  of  trying  to  fill  the  gap,  the  absence  resulted  in  additional 
problems.  One  example,  for  the  squad  the  plan  to  switch  the  simulator  type  on  the  second  day 
proved  unworkable.  The  squad  members  reverted  to  the  same  simulator  they  used  on  the  first 
day. 


RESULTS  AND  DISCUSSION 

This  section  is  organized  as  follows:  Soldier  characteristics,  simulator  capabilities  ratings, 
training  effectiveness  ratings,  interview  responses,  and  simulator  sickness  symptoms. 


Soldier  Characteristics 

Each  squad  had  nine  Soldiers  with  an  appropriate  MOS  (11  Bravo).  The  Soldiers  who  served  as 
squad  leaders  and  fire  team  leaders  during  the  VSTS  exercises  all  had  previous  experience  at 
their  respective  positions  in  a  dismounted  infantry  squad.  Time  in  service  ranged  for  6  to  78 
months,  with  an  average  of  32  months.  (The  Soldier  background  survey  is  shown  in  Appendix 
C). 

Simulator  Capabilities 

At  the  end  of  each  day  Soldiers  rated  their  capability  to  perform  each  of  62  functions  in  the 
simulator  (Appendix  D).  The  rating  scale  was  Excellent  =  5,  Very  Good  =  4,  Good  =  3,  Average 
=  2,  Poor  =  1,  Very  Poor  =  0.  (Because  one  Soldier  was  not  available  at  the  end  of  his  second 
day  the  tables  are  based  on  17,  not  18,  questionnaires.) 
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Table  1. 

Simulator  Capabilities  Ratings 


Funetion 

Combined 

Tethered 

Wearable 

Move  as  an  individual 

3.49 

3.5 

3.47 

Aim  weapon. 

3.21 

3.06 

3.33 

Move  through  open  areas  as  a  widely  separated  group. 

3.17 

3.19 

3.16 

Fire  weapon  aeeurately. 

3.17 

3.19 

3.16 

Seleet  fire  mode 

3.11 

3.13 

3.11 

Sean  from  side-to-side. 

3.11 

3.44 

2.84 

Fire  weapon  in  short  bursts. 

3.09 

3.07 

3.11 

Sean  vertieally. 

2.97 

3.06 

2.89 

Move  elose  to  walls. 

2.97 

3.13 

2.84 

Loeate  assigned  areas  of  observation,  e.g.  aeross  the 
street. 

2.94 

2.88 

3 

Assume  defensive  positions. 

2.91 

3.07 

2.79 

Take  position  to  one  side  of  the  doorway. 

2.91 

3.29 

2.63 

Move  in  single  file. 

2.89 

2.88 

2.89 

Identify  eovered  and  eoneealed  routes. 

2.88 

3.14 

2.68 

Move  aeeording  to  direetions. 

2.86 

2.88 

2.84 

Move  quiekly  to  the  point  of  attaek. 

2.83 

2.81 

2.84 

Loeate  enemy  soldiers  inside  buildings  firing  at  your 
unit. 

2.77 

2.81 

2.74 

Coordinate  with  other  squad  members. 

2.77 

3.06 

2.53 

Engage  targets  within  a  room. 

2.74 

2.88 

2.63 

Identify  enemy  soldiers. 

2.74 

2.93 

2.58 

Exeeute  planned  route. 

2.71 

2.69 

2.74 

Identify  safe  and  danger  areas. 

2.71 

3 

2.47 

Maintain  position  relative  to  other  team/squad  members. 

2.71 

3.13 

2.37 

Exeeute  the  assault  as  planned. 

2.69 

2.88 

2.53 

Identify  seetor  of  responsibility. 

2.67 

2.73 

2.61 

Identify  assigned  seetors  of  observation. 

2.66 

2.88 

2.47 

Self  awareness  of  posture 

2.65 

3.07 

2.32 

Climb  up  or  down  stairs. 

2.63 

2.88 

2.42 

Move  in  taetieal  formation 

2.63 

2.94 

2.37 

Eoeate  support  team  positions. 

2.59 

2.64 

2.56 

Identify  areas  that  mask  supporting  fires. 

2.59 

2.69 

2.53 

Employ  taetieal  hand-held  smoke  grenades. 

2.57 

2.78 

2.47 

Identify  non-eombatants  within  a  room. 

2.56 

2.63 

2.5 

Sean  the  room  quiekly  for  eombatants. 

2.56 

2.69 

2.44 

Eoeate  fire  team  buddy  positions. 

2.56 

2.73 

2.42 

Maneuver  elose  to  others. 

2.56 

2.93 

2.26 

Maneuver  below  windows. 

2.54 

2.43 

2.64 
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Table  2. 

Simulator  Capabilities  Ratings  (eontinued) 


Funetion 

Combined 

Tethered 

Wearable 

Maneuver  around  obstaeles. 

2.53 

2.81 

2.28 

Identify  eivilians/non  eombatants 

2.51 

2.75 

2.32 

Take  a  taetieal  position  while  within  a  room. 

2.51 

2.75 

2.32 

Move  past  furniture  in  a  room. 

2.49 

2.69 

2.32 

Look  around  eorners. 

2.46 

2.5 

2.42 

Take  hasty  defensive  positions. 

2.43 

2.5 

2.37 

Estimate  distanees  from  self  to  a  distant  objeet. 

2.43 

2.5 

2.37 

Maneuver  around  eorners/PIE. 

2.4 

2.44 

2.37 

Use  fragmentation  grenades. 

2.39 

2.27 

2.47 

Communieate  SPOT  reports  to  squad  leader. 

2.39 

2.64 

2.21 

Understand  verbal  eommands. 

2.38 

2.67 

2.16 

Communieate  enemy  loeation  to  team  member. 

2.34 

2.38 

2.32 

Use  flash-bang  grenades  to  help  elear  rooms. 

2.33 

2.33 

2.33 

High  erawl. 

2.31 

3.25 

2 

Move  quickly  through  doorways. 

2.29 

2.56 

2.05 

Distinguish  between  friendly  and  enemy  fire. 

2.26 

2.19 

2.32 

Maneuver  past  other  personnel  in  a  room. 

2.26 

2.44 

2.11 

Visually  locate  the  source  of  enemy  fire. 

2.26 

2.63 

1.95 

Use  hand-held  illumination  (flares). 

2.25 

2 

2.45 

Eow  crawl. 

2.24 

2.6 

2.08 

Determine  the  direction  enemy  rounds  are  coming  from. 

2.21 

2.4 

2.05 

Determine  other  team/squad  members'  positions. 

2.17 

2.63 

1.79 

Determine  the  source  of  enemy  fire  by  sound. 

1.86 

2 

1.74 

Open  gates 

1.5 

1.31 

1.67 

Open  doors 

1.41 

1.31 

1.5 

N=17 


Table  1  presents  the  average  ratings  of  the  ability  to  perform  eaeh  of  62  aetions  in  the  simulation. 
The  eolumn  labeled  “eombined”  lists  in  deseending  order  the  average  rating  eombining  both 
simulator  types.  The  next  two  eolumns  break  down  the  ratings  by  simulator  type.  The  average 
rating  for  all  items  was  2.74  for  the  tethered  systems  and  2.49  for  the  wearable.  Beeause  the 
ratings  were  similar  aeross  simulator  types  the  following  diseussion  is  based  on  the  eombined 
eolumn. 

Six  of  the  items  had  a  rating  above  3  “good”.  Two  of  these  involve  movement  and  four  involve 
use  of  the  weapon.  These  findings  are  eonsistent  with  previous  researeh  involving  similar 
simulators  (Knerr,  Garrity,  and  Lamp  ton,  2004).  Fifty-two  items  were  rated  between  “average” 
and  “good”. 

Three  items  had  average  ratings  between  “average”  and  “poor”.  These  were  “Determine  the 
souree  of  enemy  fire  by  sound.”,  “Open  gates”,  and  “Open  doors”.  The  laek  of  realistie  three 
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dimensional  sound  was  mentioned  during  the  interviews  as  a  system  shortfall.  However,  in  the 
real  world  loeation  of  enemy  fire  by  sound  in  urban  areas  is  a  well  known  ehallenge.  So  in  this 
case  the  low  rating  may  not  represent  that  big  of  a  problem. 

In  contrast,  the  low  ratings  for  opening  doors  and  gates  do  indicate  problems  that  need  to  be 
resolved.  Some  of  the  doors  in  the  VE  have  been  modeled  such  that  they  can  be  opened,  others 
cannot.  For  those  that  can  be  opened,  it  is  not  clear  to  the  users  how  to  open  the  doors.  This  led 
to  very  frustrating  episodes  in  which  the  rest  of  a  fire  team  was  yelling  at  one  team  member  who 
was  desperately  trying  to  open  a  door.  Either  the  interface  control  input  or  the  system  train-up 
need  to  be  improved  so  that  they  can  open  the  doors.  (Note:  the  point  of  this  is  not  to  train  how 
to  open  doors,  but  tactical  considerations  of  “when”  to  open  doors,  and  synchronization,  during 
room  and  building  clearing  tasks.) 

Training  Effectiveness  Ratings 

Table  2  presents  in  descending  order  the  participants’  average  individual  self-ratings  of  training 
effectiveness  for  a  set  of  tasks  highly  relevant  to  dismounted  operations  in  urban  environments. 
The  list  was  prefaced  by  the  statement:  “As  a  result  of  today’s  exercises,  my  ability  to  perform 
the  following  tasks  was  changed  as  follows”.  The  rating  scale  was 
No  improvement  =  0,  Slight  improvement  =  1,  Moderate  Improvement  =2,  and  Vast 
Improvement  =  3  (see  Appendix  E). 

The  highest  rated  task  was  “perform  voice  communications”.  The  average  rating  of  .97  falls 
slightly  below  “1”,  the  value  that  corresponds  to  “slight  improvement”.  Superficially,  this  could 
be  thought  of  as  an  unexpected  outcome  in  that  during  this  evaluation  the  intra-squad 
communication  system  was  inoperable:  the  squad  members  simply  yelled  at  each  other.  (The 
squad  leader  did  have  a  hand  held  radio  to  communicate  with  the  role  player  representing  the 
platoon  leader.)  However,  we  believe  that  this  reflects  that,  whatever  the  limitations  or 
capabilities  of  the  simulation,  the  scenarios  did  lead  to  situations  in  which  the  squad  members 
had  to  communicate  with  each  other,  and  this  practice  lead  to  improvement. 

The  next  to  lowest  rated  item  “React  to  unexploded  ordnance  hazard”  is  related  to  the  limited 
resolution  of  the  HMDs  and  the  lack  of  pilot  testing  the  scenarios.  The  Soldiers  did  not  spot  the 
hazard  before  it  exploded.  The  item  with  the  lowest  average  rating  was  “Employ  hand 
grenades”.  To  employ  grenades  the  participant  had  to  cycle  thru  a  list  of  menu  options,  select  a 
type  of  grenade,  and  then  use  a  power  bar  approach  and  the  angle  of  the  hand  held  weapon  to 
determine  the  trajectory  of  the  grenade. 
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Table  3. 

Perceived  Training  Effectiveness 


Activity 

Rating 

Perform  voice  communications 

0.97 

React  to  contact 

0.79 

Conduct  a  combat  patrol 

0.76 

Engage  targets  during  an  urban  operation 

0.74 

Move  as  a  member  of  a  fire  team 

0.74 

React  to  Snipers 

0.70 

Conduct  Tactical  Movement  in  an  urban  area 

0.70 

React  to  direct  fire 

0.69 

Perform  movement  techniques  during  an  urban  operation 

0.67 

Assault  a  building 

0.66 

Move  over,  through,  or  around  obstacles 

0.65 

Move  under  direct  fire 

0.61 

React  to  indirect  fire 

0.60 

Search  a  building 

0.59 

Enter  a  building  during  an  urban  operation 

0.58 

Engage  targets  with  assigned  weapon 

0.57 

Avoid  an  Ambush 

0.57 

Select  temporary  fighting  position 

0.53 

Break  contact 

0.42 

React  to  a  civil  disturbance 

0.38 

Secure  at  a  halt 

0.34 

Establish  an  observation  post 

0.29 

React  to  unexploded  ordnance  hazard 

0.26 

Employ  hand  grenades 

0.24 

N=  17 


The  training  effectiveness  questionnaire  contained  24  unique  items.  The  item  “react  to  indirect 
fire”  inadvertently  appeared  twice  on  the  printed  list,  separated  by  5  other  items.  Each 
occurrence  of  “react  to  indirect  fire”  had  the  same  mean  rating,  which  provides  some  evidence 
that  the  participants  were  actually  reading  the  items  and  responding  consistently. 

In  general  these  ratings  of  training  effectiveness  are  low.  We  believe  the  ratings  are  a  function  of 
several  factors.  Execution  of  well  designed  scenarios  was  hindered  by  technical  difficulties  that 
affected  both  the  conduct  of  the  exercises  and  the  AAR.  During  previous  research  using 
somewhat  similar  VE  systems  at  the  Soldier  Battle  Eab  at  Et.  Benning,GA,  and  the  Cassidy 
MOUT  site  at  Et.  Campbell,  KY,  we  observed  better  technical  execution  of  exercises  that  in  turn 
led  to  more  in-depth  AARs  (Knerr  &  Eampton,  2005). 
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There  may  be  another  eritieal  faetor  that  had  an  impact  on  the  ratings.  Many  of  the  Soldiers 
indicated  during  the  interviews  that  they  already  knew  how  to  perform  these  tasks  before  they 
took  part  in  this  evaluation. 

Interviews 

For  each  question,  the  interview  results  for  the  squad  leader  and  fire  team  leaders  are  presented 
first,  then  “S-“  denotes  the  comments  from  the  other  squad  members. 

During  the  two  days  you  were  here  you  used  two  different  types  of  simulators  (tethered  and 
wearable).  Did  you  find  you  could  do  some  things  better  in  one  type  of  simulator  than  in 
the  other?  If  so,  what  could  you  do  better  in  the  tethered  simulator? 

•  Walk  in  a  straight  line. 

•  Movement  was  better  because  the  tethered  simulator  was  more  responsive  and  had  less 
lag. 

•  S  -  The  tethered  system  was  lighter,  the  visual  display  was  better,  the  system  reacted 
quicker,  and  weapon  worked  better. 

What  could  you  do  better  in  the  wearable  simulator? 

•  Head  movement  was  easier. 

•  The  wearable  simulator  provided  free  range  of  motion. 

•  S  -  The  strengths  of  the  wearable  system:  helmet  more  comfortable,  unrestricted 
movement,  lifelike. 

Did  you  find  some  things  were  easier  to  do  in  one  type  of  simulator  than  in  the  other?  If  so, 
what  was  easier  in  the  tethered  simulator? 

•  In  contrast  to  the  wearable  system  “Didn’t  need  two  people  to  put  on  the  tethered  system” 

What  was  easier  in  the  wearable  simulator? 

•  Nothing  mentioned 

Did  you  feel  more  comfortable  in  one  type  of  simulator  than  in  the  other?  If  so,  which 

•  The  HMD  for  the  tethered  system  is  uncomfortable  (this  was  restated  several  times).  It 
needs  padding.  It  needs  an  eye  cup  to  avoid  distractions.  The  cable  for  the  tethered 
system  gets  “stiff’  during  turns  and  needs  “slack”.  The  tethered  system  is  more  of  a 
problem  for  very  tall  users. 

•  The  wearable  system  is  hotter  than  the  tethered  system.  The  leg  strap  (which  held  in  place 
a  position  sensor)  on  the  wearable  system  was  the  source  of  several  complaints  related  to 
discomfort  and  problems  with  walking  in  a  straight  line  in  the  Virtual  Environment. 
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•  In  the  wearable  the  view  gets  stuek  in  the  “down”  position  making  you  look  at  the 
ground.  To  look  up,  you  must  keep  the  weapon  up,  this  in  turn  leads  to  arm  fatigue. 

•  Looking  up  while  prone  was  a  problem  in  both  systems. 

•  The  wearable  simulator  requires  too  many  adjustments,  but  the  wearable  head  mount  is 
better. 

•  If  the  fit  of  the  HMD  on  the  tethered  were  better,  then  the  tethered  would  be  clearly 
better.  No  leg  strap,  better  focus,  can  stand  naturally,  just  works  better.  The  Army  is  not 
about  comfort,  so  even  if  the  HMD  fits  better  on  the  wearable  the  tethered  is  clearly  a 
better  choice. 

What  did  you  like  most  about  the  scenarios? 

•  Realistic  environments:  buildings,  people,  trash,  lack  of  vegetation  added  to  realism. 

•  Resembled  Iraq.  Was  realistic  in  the  expanse  of  the  virtual  environment,  not  just  a 
MOUT  area  with  two  buildings.  Freedom  to  patrol  where  you  wished.  Room  to 
maneuver. 

•  Not  physically  stressful  (this  is  both  good  and  bad  from  a  training  perspective). 

•  S  -  The  good  points  of  the  scenarios:  Engage  targets,  practice  reactions  to  events, 
identification  of  people,  practice  ready-up,  lane  of  fire,  practice  mind  -  not  body. 

What  did  you  like  least  about  the  scenarios? 

•  OPFOR  didn’t  shoot,  wouldn’t  move,  just  stood  there 

•  About  the  only  ways  to  interact  with  suspects  was  to  ignore  them  or  shoot  them,  that  is, 
they  could  not  detain  suspects. 

•  Insurgents  were  always  dressed  the  same  way 

•  The  simulation  of  vehicles  (traffic)  was  not  good.  Hard  to  identify  objects.  Can’t  see 
people  at  a  distance. 

•  S  -  The  weak  points  of  the  scenarios:  not  enough  enemies  to  shoot,  restricted,  unrealistic 

What  part  of  the  simulation  (tasks,  conditions,  buildings,  terrain,  etc.)  was  the  most 
realistic? 

•  Trash  all  over  was  good, 

•  The  environment,  variety  of  buildings,  not  just  a  perfect  house  without  function. 

•  Could  actually  run  in  and  assault  the  building. 

•  Room  to  travel  a  great  distance  using  complicated  routes 

What  part  of  the  simulation  (tasks,  conditions,  buildings,  terrain,  etc.)  was  the  least 
realistic? 

•  Should  add  dogs  and  little  kids. 

•  Should  have  sun  set  or  night  operations. 

•  Sound  of  indirect  fire  was  not  realistic. 
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•  When  walking  on  road  there  is  no  peripheral  vision,  so  you  ean  pass  by  a  street  without 
recognizing  it  as  a  street.  You  must  turn  directly  and  look  at  it. 

•  Hand  and  arm  signals  take  too  long.  You  need  to  be  able  to  point  at  stuff,  but  by  the  time 
it  works  it’s  too  late. 

•  S  -  There  are  no  lEDs  and  no  weapon  malfunctions. 

What  was  the  most  difficult  task  to  perform  in  the  simulator? 

•  Enter  and  clear  a  room 

•  Differentiate  team  members.  (This  was  addressed  for  Squad  #  2  by  having  an  actual 
name  or  nickname  appear  on  each  avatar.) 

•  Voice  communications  (need  for  the  head  sets  to  work). 

•  Difficult  to  maneuver  close  to  objects,  negotiate  doors,  pass  by  a  squad  member. 

•  Difficult  to  control  team  and  keep  track  of  where  team  is. 

•  Hard  to  keep  weapon  up  (big  problem). 

•  S  -  Difficult  to  open  and  get  into  doors,  difficult  to  use  menu  for  so  many  tasks,  difficult 
to  determine  who  is  who/who  did  what 

Did  you  find  any  aspects  of  the  simulator  or  simulation  distracting? 

•  How  to  use  properly:  need  at  least  an  entire  day  just  to  learn  the  system.  (This  was 
restated  several  times.). 

•  You  need  to  learn  self  troubleshooting  for  the  simulator. 

•  Some  times  when  you  are  close  to  a  wall  you  can  see  through  the  wall. 

•  You  can  go  through  some  walls. 

•  People  just  show  up  (teleport). 

•  Your  view  changes  45  degrees  when  your  weapon  comes  up. 

•  S  -  the  tether,  glare  on  HMD,  accidental  activation  of  calibration  button  on  weapon,  in 
wearable:  bumping  into  things  and  people 

Do  you  think  that  the  simulators  you  used  today  were  a  useful  training  or  sustaining  tool 
(that  is,  did  you  learn  from  today’s  experience  or  could  it  he  used  to  train  or  sustain 
comhat  skills)? 


•  Not  a  substitute  for  physically  training  in  the  field 

•  They  already  knew  how  to  do  lead  a  team  before  the  virtual  exercises 

•  Made  them  think 

•  Has  potential,  but  need  to  train  with  it  for  more  than  2  days 

•  Pretty  good  set  up  but  needs  more  realistic  movement. 

•  Should  include  Stryker  and  convoy  exercises. 

•  If  perfected  (without  bugs)  it  could  be  used  to  train  newer  people,  but  not  for  sustaining 
skills  for  experienced  personnel. 

•  Could  walk  new  people  though  procedures  in  a  safe  environment. 
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•  Would  not  use  after  live  fire,  would  be  going  baekwards. 

•  Good  for  basie  training,  but  would  need  four  days  to  be  eomfortable  with  the  system  and 
the  trainees  must  take  it  seriously. 

•  Maybe  good  for  taetieal  formations  and  patrols,  eall  for  fire,  med  evaeuation,  reaet  to 
ambush,  but  not  for  room  elearing  and  not  for  lED 

•  Non-combatants:  react,  take  cover,  return  fire,  call  higher  (radio  the  Platoon  Leader) 

What  did  you  learn  or  how  could  it  he  used  to  train  or  sustain? 

•  Trained  =  Combat  patience 

•  Sustained  =  movement  techniques 

Could  the  simulators  used  to  train  non-comhat  arms  Soldiers? 

•  Could  help  train  drivers,  but  might  give  false  sense  of  security  to  non-combat  arms 
Soldiers 

Do  you  think  that  this  training  will  affect  your  performance  in  a  real  world  setting? 

•  Resounding  “No!”  Might  help  inexperienced  trainees  if  they  spent  some  time  in  it. 

•  The  situation  dictates.  It  might  help  in  some  settings  and  not  in  others 

•  If  you  have  already  been  there  (in  combat)  won’t  help 

•  If  you  have  already  been  to  Iraq  two  or  three  times,  it  will  n^  help. 

Where  in  a  unit  training  program  do  you  think  that  this  type  of  training  would  he  most 
appropriate  or  useful? 

•  Could  help  every  one/any  one  for  reaction  to  contact. 

•  Could  help  inexperienced  leaders:  LTs,  PL  SGT,  maybe  SL 

•  Lower  enlisted  personnel,  just  basic  things 

•  Could  be  used  for  introduction  for  new  Soldiers 

•  Does  not  replace  any  other  training 

•  S  -  Good  for  basic  training,  communication  training,  visual  search  of  threats 

•  S  -  This  training  could  improve  reactions  in  the  real  world  and  build  knowledge  on  what 
to  do,  but  might  cause  someone  to  think  it  is  just  a  game:  no  consequences. 

•  S  -  This  type  of  training  would  be  good  for  crowd  control  training,  new  skills,  and 
mission  rehearsal. 

Under  what  circumstances  or  conditions  would  you  use  this  type  of  capability  in  a  field 
environment  (for  example  —  an  assembly  area  to  conduct  a  rehearsal  for  an  actual 
operation;  to  train  on  new  skills  during  training  events;  to  practice  unit  operations;  etc)? 
("Never"  is  an  acceptable  answer.) 

•  Could  be  used  for  all  the  examples  given  above  if  hadn’t  been  in  the  field  for  awhile. 
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•  Could  come  to  the  BCTC  as  a  platoon,  do  squad  exercises,  then  platoon  exercise. 

•  If  you  have  a  virtual  model  of  the  aetual  area  would  help  rehearsal  and  Military 
Intelligence,  would  be  useful  as  a  high  tech  sand  table. 

If  you  believe  the  wearable  computer  simulation  has  merit  for  field  use,  what  would  have  to 
be  changed  about  the  current  system  to  make  it  more  usable  in  the  field  environment? 

•  (Reacted  with  laughter  to  this  question)  Make  it  waterproof,  use  it  in  a  trailer  not  a 
rucksack  approach 

•  Leg  stuff  (sensor  on  the  leg)  is  not  good 

•  Need  better  sound  (direetional  sound) 

•  Need  commo  within  the  team 

•  Other;  avatars  were  running  (instead  of  walking)  all  the  time  need  little  bar  that  says  you 
are  tired  (as  in  many  game  systems),  frustrating  because  you  are  running  (when  you  want 
to  walk)  and  not  running  straight 

•  S  -  To  use  in  the  field,  the  system  would  need  to  be  more  durable  and  rugged. 

What  is  your  overall  reaction  to  the  VSTS? 

•  Has  potential  but  needs  a  lot  more  work 

•  When  fixed,  give  users  more  praetice 

•  Could  be  used,  but  should  not  be  relied  on 

•  Could  be  useful  if  refined  (fixed) 

Simulator  Sickness 

Sickness  from  simulator  exposure  was  measured  using  a  modified  version  (see  Appendix  E)  of 
the  Simulator  Sickness  Questionnaire  (SSQ)  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993). 

One  item  was  added  to  differentiate  between  eold  sweating,  thought  to  be  related  to  simulator 
exposure,  and  natural  warm  sweating.  Warm  sweating  is  related  to  normal  sweating  from 
physical  exertion;  ambient  room  temperature;  or  type  of  clothing,  harness,  or  wearable 
equipment.  Since  warm  sweating  is  not  thought  to  be  related  to  the  negative  side  effects  of  the 
simulator  itself,  it  was  not  ineluded  in  the  subseale  and  total  severity  scoring.  Items  were  rated 
by  participants  as  either  none,  slight,  moderate,  or  severe  (later  scored  by  the  experimenter  as  0, 

1,  2,  and  3  respeetively).  These  raw  rating  scores  were  used  in  the  calculations  as  suggested  in 
Knerr  et  al.  (1998)  instead  of  the  weighted  scores  suggested  in  Kennedy  et  al.  (1993). 
Combinations  of  scored  items  summarize  three  distinct  symptom  clusters,  including  (a)  nausea 
(stomaeh  awareness,  increased  salivation,  burping),  (b)  oeulomotor  (eye  strain,  headache,  blurred 
vision,  difficulty  focusing),  and  (c)  disorientation  (dizziness,  vertigo).  The  combination  of  the 
three  symptom  clusters  summarize  the  total  severity  of  sickness  experienced.  Planned  paired 
comparisons  were  performed  for  change  scores  for  each  of  the  subscale  scores,  as  well  as  total 
severity  score.  An  alpha  level  of  .05  was  used  for  all  statistical  tests. 
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Table  3. 

Changes  in  Reported  Symptoms  as  a  Result  of  Simulator  Use 


All  Configurations 

N 

Pre 

Post 

Change 

p  Value 

Total  Severity 

36 

0.49 

3.33 

2.85 

0.00 

Nausea 

36 

0.17 

0.67 

0.50 

0.03 

Oeulomotor 

36 

0.24 

1.89 

1.65 

0.00 

Disorientation 

36 

0.08 

0.78 

0.69 

0.02 

Wearable 

N 

Pre 

Post 

Change 

p  Value 

Total  Severity 

20 

0.75 

2.5 

1.75 

0.02 

Nausea 

20 

0.25 

0.5 

0.25 

0.33 

Oeulomotor 

20 

0.35 

1.5 

1.15 

0.01 

Disorientation 

20 

0.15 

0.5 

0.35 

0.03 

Tethered 

N 

Pre 

Post 

Change 

p  Value 

Total  Severity 

16 

0.16 

4.38 

4.22 

0.01 

Nausea 

16 

0.06 

0.88 

0.81 

0.06 

Oeulomotor 

16 

0.09 

2.38 

2.28 

0.00 

Disorientation 

16 

0.00 

1.13 

1.13 

0.01 

As  ean  be  seen  in  Table  3  above,  overall,  total  severity  seores  and  all  sub-seale  seores  were 
signifieantly  inereased  after  simulator  exposure.  Signilieant  inereases  in  nausea  were  not  related 
to  simulator  use  in  the  wearable  eonfiguration.  This  eould  have  been  due  to  a  number  of  reasons. 
Although  the  tethered  system  had  a  higher  resolution  HMD,  it  was  less  eomfortable  and 
eompletely  oeeluded  visual  stimuli  from  the  real  world.  Conversely,  the  wearable  eonfiguration 
was  reported  by  the  Soldiers  as  very  eomfortable  (probably  beeause  it  was  retro-fitted  into  the 
eombat  helmet  they  are  very  used  to  wearing).  The  wearable  HMD  did  not  oeelude  the  real- 
world  peripheral  stimuli  to  the  same  degree  as  the  tethered  system.  Visual  stimuli  from  the  real- 
world  may  have  provided  motion  ones  that  aided  the  Soldier  in  avoiding  siekness  eaused  by 
artifieial  movement  seen  solely  from  a  eompletely  immersive  HMD.  A  one-way  ANOVA  was 
performed  to  determine  if  the  post  seores  were  signifieantly  different  between  the  wearable  and 
tethered  systems.  As  Table  4  summarizes,  no  signifieant  differenees  were  observed. 
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Table  4. 

ANOVA  Table  of  Differenees  Between  Wearable  and  Tethered  Post-Simulation  Siekness  Scores 


Sum  of 
Squares 

df 

Mean 

Square 

F 

p  Value 

Total  Severity 

Between  Groups 

31.25 

1 

31.25 

1.83 

0.19 

Within  Groups 

580.75 

34 

17.08 

Total 

612.00 

35 

Nausea 

Between  Groups 

1.25 

1 

1.25 

0.91 

0.35 

Within  Groups 

46.75 

34 

1.38 

Total 

48.00 

35 

Oculomotor 

Between  Groups 

6.81 

1 

6.81 

1.52 

0.23 

Within  Groups 

152.75 

34 

4.49 

Total 

159.56 

35 

Disorientation 

Between  Groups 

3.47 

1 

3.47 

2.42 

0.13 

Within  Groups 

48.75 

34 

1.43 

Total 

52.22 

35 

Table  5. 

Changes  in  Individual  Symptoms  as  a  Result  of  Simulator  Use 


N 

Pre 

Post 

Difference 

p  Value 

Vertigo 

18 

0 

0 

0 

Burping 

18 

0 

0 

0 

Salivation  Increased 

18 

0.03 

0 

-0.03 

0.33 

Nausea 

18 

0.06 

0.11 

0.06 

0.43 

Difficulty  Concentrating 

18 

0.03 

0.08 

0.06 

0.43 

Cold  Sweating 

18 

0.03 

0 

-0.03 

0.33 

Dizzy  eyes  closed 

18 

0 

0.03 

0.03 

0.33 

Stomach  aware 

18 

0 

0.06 

0.06 

0.33 

"Full  head" 

18 

0 

0.14 

0.14 

0.24 

Dizzy  eyes  open 

18 

0 

0.17 

0.17 

0.11 

Blurred  Vision 

18 

0 

0.11 

0.11 

0.10 

Fatigue 

18 

0 

0.25 

0.25 

0.06 

Headache 

18 

0.06 

0.33 

0.28 

0.04 

General  Discomfort 

18 

0.06 

0.36 

0.31 

0.04 

Difficulty  Focusing 

18 

0.03 

0.33 

0.31 

0.04 

Eyestrain 

18 

0.08 

0.53 

0.44 

0.00 

Four  individual  symptoms  significantly  increased  after  simulator  exposure:  general  discomfort, 
headache,  difficulty  focusing,  and  eyestrain,  which  had  the  largest  increase  over  the  other 
symptoms,  as  summarized  in  Table  5  above.  Interestingly,  these  are  all  oculomotor  symptoms 
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and  are  considered  mild  sickness  symptoms;  the  more  severe  symptoms  (e.g.,  dizzy,  vertigo, 
stomach  awareness,  burping)  had  negligible  change  scores  and  were  non-significant. 

One  particular  simulator  station,  not  type  of  simulator  but  a  specific  simulator  station,  seemed  to 
be  associated  with  elevated  symptom  scores.  That  simulator  was  flagged  for  subsequent 
inspection. 


SUMMARY  AND  CONCLUSIONS 

Frequent  technical  problems  with  the  system  and  some  of  the  individual  simulators  interfered 
with  conducting  the  evaluation  as  planned.  More  importantly,  technical  problems  undoubtedly 
influenced  almost  every  aspect  of  the  Soldiers’  ratings  of  the  system  components,  and  the  system 
as  a  whole. 

The  self-ratings  of  the  ability  to  perform  62  actions  in  the  simulator  were  consistent  with 
previous  research  involving  similar  simulators.  Six  of  the  items  had  a  rating  above  3  “good”. 

Two  of  these  involve  movement  and  four  involve  use  of  the  weapon.  Fifty-two  items  were  rated 
between  “average”  and  “good”.  Three  items  had  average  ratings  between  “average”  and  “poor”. 
These  were  “Determine  the  source  of  enemy  fire  by  sound.”,  “Open  gates”,  and  “Open  doors”. 

The  participants’  average  individual  self-ratings  of  training  effectiveness  for  a  set  of  24  tasks 
highly  relevant  to  dismounted  operations  in  urban  environments  were  low.  The  average  for  the 
highest  rated  task  fell  slightly  below  “1”  the  value  that  corresponds  to  “slight  improvement”. 

Interviews  identified  strengths  and  weaknesses  of  the  system.  Unlike  previous  research  there 
were  few  comments  about  the  need  to  add  functions  to  the  system.  Rather,  the  functions  that  are 
already  there  need  to  work  better.  Throughout  the  interview,  regardless  of  the  specific  questions, 
there  were  frequent  comments  that  the  system  was  frustrating  to  use,  that  the  “bugs”  should  be 
fixed,  and  more  practice  was  needed  on  how  to  use  the  system. 

Compared  to  previous  research  with  HMDs,  the  frequency  and  severity  of  simulator  sickness 
symptoms  were  very  low.  No  one  withdrew  from  the  evaluation  because  of  simulator  sickness. 
The  predominant  symptoms  were  related  to  eye  strain.  There  were  no  major  problems  with 
nausea  or  dizziness,  which  are  the  symptom  categories  that  are  more  problematic  from  a  safety 
perspective. 

It  is  unfortunate  that  the  necessary  modifications  to  the  network  and  simulators  were  not  made 
before  the  resources  of  this  evaluation  were  expended.  If  system  performance  is  not  going  to  be 
improved  over  the  level  we  observed,  then  the  usefulness  of  the  system  is  in  question  because 
training  effectiveness  would  be  compromised  by  the  various  system  problems. 
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APPENDIX  A:  THE  VIRTUAL  SQUAD  TRAINING  SYSTEM 


VSTS  consists  of  four  immersive  tethered  Soldier  systems,  five  immersive  wearable 
Soldier  systems,  a  BattleMaster  station,  a  Semi-Automated  Force  (SAP)  workstation,  a  Scenario 
Authoring  station  and  an  After  Action  Review  station.  The  VSTS  weapons  mix  includes:  5  x 
M4,  2  X  M16/M203  and  2  x  M249.  VSTS  has  six  terrain  databases  available  for  training, 
including  a  geo-typical  database  of  Northern  Baghdad,  Iraq.  A  tethered  system  uses  a  light 
weight  cable  to  connect  to  the  appropriate  computing  system  and  the  Soldier’s  movement  is 
somewhat  restricted.  A  wearable  system  uses  a  wireless  lightweight  durable  computer  that  is 
attached  to  the  Soldier’s  back  and  movement  is  not  restricted. 


Figure  4  A-1 .  Virtual  squad  training  system. 
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APPENDIX  B;  DAILY  SCHEDULES 


0730 

0800 


0835 


1030 

1045 


1205 

1330 


1430 


1550 


1630 

1700-U/C 


Each  Soldier  uses  two  simulators  over  a  two-day  period. 

May  8/10  2007 


Set-up  -  All 
Soldiers  Arrive 

•  Training  Assessment  Briefing  -  TPIO  Virtual 

•  Symptom  Cheeklist  -  ARI 

Simulator  Orientation/Calibration  -  TPIO  Virtual/Instruetor  Operators  (lO) 

•  Suit  up/Calibrate 

•  Move  and  shoot  training  exereise 

•  Move,  shoot  &  eommunieate  training  exereise 

•  Foree  on  Foree 
Break 

Reaet  to  Sniper  (Buddy  teams) 

•  Reeeive  plan  -  (10  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (4  buddy  teams)  (40  minutes) 

•  AAR  (  20  minutes)  Note:  all  AARs  inelude  administering  symptom  eheeklist 
Luneh 

Identify/reaet  to  lED  (Squad) 

•  Reeeive  orders  and  plan  -  (10  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (20  minutes) 

Break  Contaet  (Squad) 

•  Reeeive  orders  and  plan  -  (20  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (20  minutes) 

Questionnaires  &  Interviews  —  ARI 

•  Symptom  Cheeklist 

•  Simulator  Performanee  Questionnaire 
Release  Soldiers 

Administrative  AAR/Readjust  plan 
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May  9/1 1  2007 


0730  Set-up -All 

0830  Soldiers  Arrive 

•  Symptom  Cheeklist  —  ARI 

0840  Familiarization  Exereises 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  AAR  (10  minutes) 

0920  STX  Reaet  to  Ambush  (Squad) 

•  Review  plan  (10  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (20  minutes) 

1040  Training  Exereise  -  Dismounted  Combat  Patrol 

•  Reeeive  orders  and  plan  -  (20  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (  20  minutes) 

1200  Luneh 

1300  Training  Exereise  -  Combat  PatroFClear  a  building/room 

•  Reeeive  orders  and  plan  -  (20  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (20  minutes) 

1420  Training  Exereise  -  Combat  Patrol/Call  for  ORF/MEDIVAC 

•  Reeeive  orders  and  plan  -  (20  minutes) 

•  Move  to  simulators/ealibrate  weapons  (10  minutes) 

•  Conduet  exereise  (20  minutes) 

•  Break  (10  minutes) 

•  AAR  (20  minutes) 

1540  Questionnaires  &  Interviews  —  ARI 

•  Symptom  Cheeklist 

•  Simulator  Performanee  Questionnaire 

•  Group  Interviews 

1630  Release  Soldiers 
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APPENDIX  C:  BIOGRAPHICAL  INFORMATION  QUESTIONNAIRE 


Soldier’s  Name _ 

Date _ 

Please  fill  in  the  blank  or  mark  or  circle  the  appropriate  response. 

1 .  What  is  your  age?  _ Y ears 

2.  MOS _ 

3.  Rank _ 

4.  Time  in  serviee:  Years _  Months _ 

4.  a.  Combat  Experience:  Years _  Months _  Tours _ 

5.  What  is  your  current  duty  position? _ 

How  long  in  this  position?  _ 

6.  What  Army  training  courses  have  you  completed?  Check  all  that  apply. 

_ OSUT/AIT  _  PLDC/WLC  _ BNCOC 

_ Airborne  _  Combat  Life  Saver  Course 

_ Air  Assault  _ Ranger  _ Other  (specify) 

7.  How  susceptible  to  motion  or  car  sickness  do  you  feel  you  are? 

1  2  3  4  5  6  7 

very  average  very 

mildly  highly 

8.  Do  you  have  normal  or  corrected  to  normal  20/20  vision?  Yes  No 

9.  Are  you  color  blind?  Yes  No 
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10.  Are  you  _  right  handed?  _ left  handed? 

11.  How  would  you  deseribe  your  level  of  eonfidence  in  using  computers? 

1  2  3  4  5 

low  average  high 

12.  How  many  hours  per  week  do  you  use  computers?  _ hours  per  week 

13.  How  many  hours  per  week  do  you  play  computer  or  video  games  (X-Box,  Playstation, 

etc.)?  _ hours  per  week 

14.  How  often  have  you  trained  at  a  MOUT  site  since  basic  training  {NOT  including 
demonstrations)? 

_  not  since  basic  training  _  1-3  times  _ more  than  3  times 

15.  Have  you  ever  participated  in  close  quarter  combat  (room  clearing)  training  or  in  combat? 

Yes  No 

16.  Have  you  ever  participated  in  a  demonstration  at  a  MOUT  site? 

Yes  No 

17.  Have  you  ever  been  in  a  Virtual  simulator  before? 

Yes  No 

18.  Have  you  had  any  other  experience  with  military  computer  simulations? 

Yes  No 

If  yes,  please  describe  briefly  or  give  the  names  of  the  simulators. 
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APPENDIX  D;  SIMULATOR  PERFORMANCE  QUESTIONNAIRE 


Simulator  Performance  Questionnaire 

Soldiers  Name:  Today’s  Date: 

Type  of  simulator:  Tethered  Wearable 

Please  rate  your  ability  to  perform  each  action  in  the  simulation 

Excellent 

Very 

Good 

Good 

Average 

Poor 

Very 

Poor 

1.  Moye  through  open  areas  as  a  widely 
separated  group. 

2.  Moye  according  to  directions. 

3.  Moye  as  an  indiyidual 

4.  Moye  in  tactical  formation 

5.  Maneuyer  around  obstacles. 

6.  Moye  in  single  file. 

7.  Maneuyer  below  windows. 

8.  Maneuyer  close  to  others. 

9.  Determine  other  team/squad  members' 
positions. 

10.  Maintain  position  relatiye  to  other 
team/squad  members. 

11.  Maneuyer  around  corners/PIE. 

12.  Locate  assigned  areas  of  obseryation, 
e.g.  across  the  street. 

13.  Look  around  corners. 

14.  Visually  locate  the  source  of  enemy  fire. 

15.  Determine  the  source  of  enemy  fire  by 
sound. 

16.  Distinguish  between  friendly  and  enemy 
fire. 

17.  Identify  ciyilians/non  combatants 

18.  Communicate  enemy  location  to  team 
member. 

19.  Take  hasty  defensiye  positions. 

20.  Aim  weapon. 

21.  Fire  weapon  in  short  bursts. 

22.  Select  fire  mode 

23.  Fire  weapon  accurately. 

24.  Identify  coyered  and  concealed  routes. 

25.  Identify  areas  that  mask  supporting  fires. 
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Simulator  Performance  Questionnaire  (Cont’d): 

Please  rate  your  ability  to  perform  each  action  in  the  simulation 

Excellent 

Very 

Good 

Good 

Average 

Poor 

Very 

Poor 

26.  Coordinate  with  other  squad  members. 

27.  Execute  the  assault  as  planned. 

28.  Move  quickly  to  the  point  of  attack. 

29.  Assume  defensive  positions. 

30.  Identify  safe  and  danger  areas. 

31.  Locate  support  team  positions. 

32.  Locate  fire  team  buddy  positions. 

33.  Take  position  to  one  side  of  the  doorway. 

34.  Move  quickly  through  doorways. 

35.  Take  a  tactical  position  while  within  a 
room. 

36.  Scan  the  room  quickly  for  combatants. 

37.  Engage  targets  within  a  room. 

38.  Open  doors 

39.  Open  gates 

40.  Identify  non-combatants  within  a  room. 

41.  Move  past  furniture  in  a  room. 

42.  Maneuver  past  other  personnel  in  a  room. 

43.  Understand  verbal  commands. 

44.  Identify  sector  of  responsibility. 

45.  Communicate  SPOT  reports  to  squad 
leader. 

46.  Execute  planned  route. 

47.  Identify  assigned  sectors  of  observation. 

48.  Move  close  to  walls. 

49.  Scan  from  side-to-side. 

50.  Scan  vertically. 

51.  Identify  enemy  soldiers. 

52.  Estimate  distances  from  self  to  a  distant 
object. 

53.  Climb  up  or  down  stairs. 

54.  Locate  enemy  soldiers  inside  buildings 
firing  at  your  unit. 

55.  Determine  the  direction  enemy  rounds 
are  coming  from. 

56.  Use  fragmentation  grenades. 

57.  Use  hand-held  illumination  (flares). 

58.  Use  flash-bang  grenades  to  help  clear 
rooms. 
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Simulator  Performance  Questionnaire  (Cont’d): 

Please  rate  your  ability  to  perform  each  action  in  the  simulation 

Excellent 

Very 

Good 

Good 

Average 

Poor 

Very 

Poor 

59.  Employ  tactical  hand-held  smoke 
grenades. 

60.  Low  crawl. 

61.  High  crawl. 

62.  Self  awareness  of  posture 
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APPENDIX  E:  TRAINING  EEEECTIVENESS 


Soldier  Questionnaire  II:  Training  Effectiveness 

ID  Number;  Today's  Date: 

Position  Heid  During  Today's  Exercise  {Check  one): 

Sauad  Leader 

Aioha  Team  Leader 

Bravo  Team  Leader 

Team  Member 

As  a  result  of  today's  exercises,  my 
ability  to  perform  the  following  tasks  was 
changed  as  follows. 

No 

Improve¬ 

ment 

Slight 

Improve¬ 

ment 

Moderate 

Improve¬ 

ment 

Vast 

Improve¬ 

ment 

Engage  targets  with  assigned  weapon 

Employ  hand  grenades 

Perform  voice  communications 

Perform  movement  techniques  during  an 
urban  operation 

Engage  targets  during  an  urban 
operation 

Enter  a  building  during  an  urban 
operation 

Move  as  a  member  of  a  fire  team 

Move  over,  through,  or  around  obstacles 

Move  under  direct  fire 

React  to  indirect  fire 

React  to  direct  fire 

React  to  unexploded  ordnance  hazard 

Select  temporary  fighting  position 

React  to  contact 

Avoid  an  Ambush 

React  to  indirect  fire 

Break  contact 

Secure  at  a  halt 

Conduct  a  combat  patrol 

React  to  Snipers 

Conduct  Tactical  Movement  in  an  urban 

area 

Assault  a  building 

Search  a  building 

Establish  an  observation  post 

React  to  a  civil  disturbance 
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APPENDIX  F:  SIMULATOR  SICKNESS  QUESTIONNAIRE 

Symptom  Checklist 

Date _  Name _ 

Time _  Tethered _ Wearable _ 

Instructions:  Please  indicate  the  severity  of  symptoms  that  apply  to  you  right  now  by  circling  the 
appropriate  word. 


1 .  General  discomfort 

None 

Slight  Moderate  Severe 

2.  Fatigue  (Vest,  Weapon) 

None 

Slight  Moderate  Severe 

3.  Headache 

None 

Slight  Moderate  Severe 

4.  Eye  Strain 

None 

Slight  Moderate  Severe 

5.  Difficulty  focusing  (HMD) 

None 

Slight  Moderate  Severe 

6.  Salivation  increased 

None 

Slight  Moderate  Severe 

7.  a.  Warm  Sweating  (from 

None 

Slight  Moderate  Severe 

temperature  or  exertion) 

b.  Cold  Sweating  (from 

None 

Slight  Moderate  Severe 

discomfort  or  nervousness) 

8.  Nausea 

None 

Slight  Moderate  Severe 

9.  Difficulty  concentrating 

None 

Slight  Moderate  Severe 

10.  "Fullness  of  the  Head" 

None 

Slight  Moderate  Severe 

11.  Blurred  Vision 

None 

Slight  Moderate  Severe 

12.  a.  Dizziness  with  eyes  open 

None 

Slight  Moderate  Severe 

b.  Dizziness  with  eyes  closed 

None 

Slight  Moderate  Severe 

13.  Vertigo 

None 

Slight  Moderate  Severe 

14.  *Stomach  awareness 

None 

Slight  Moderate  Severe 

15.  Burping 

None 

Slight  Moderate  Severe 

16.  Other  (describe): _ 

*  Stomach  awareness  is  usually  used  to  indicate  a  feeling  of  discomfort,  which  is  just  short  of 
nausea. 
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